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Abstract 

We consider statistical methods for reduction of multivariate dimensionality 
that have invariance and/or commutativity properties under the affine group 
of transformations (origin translations plus linear combinations of coordinates 
along initial axes). The methods discussed here differ from traditional principal 
component and coordinate approaches in that they are origin-centric. Because 
all Cartesian coordinates of the origin are zero, it is the unique fixed point for 
subsequent linear transformations of point scatters. Whenever visualizations 
allow shifting between and/or combining of Cartesian and polar coordinate 
representations, as in Biplots, the location of this origin is critical. Specifically, 
origin-centric visualizations enhance the psychology of graphical perception by 
yielding scatters that can be interpreted as Dyson swarms. The key factor is 
typically the analyst’s choice of origin via an initial “centering” translation; 
this choice determines whether the recovered scatter will have either no points 
depicted as being near the origin or else one (or more) points exactly coincident 
with this origin. 

Keywords: Affine Transfomations, Oblique and Orthogonal Pro¬ 
jections, Principal Components and Coordinates, Multidimensional 
Scaling, Euclidean distance, Cartesian and polar coordinates. 
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1 Introduction 

We discuss an intuitive alternative to classical Principal Components analysis that se¬ 
lects the (singular) affine transformation of the original scatter, visualized as N points 
embedded within a p-dimensional Euclidean space, that reduces its dimensionality to 
q < p while attempting to reproduce all pair-wise, squared Mahalanobis inter-point 
distances. We show that this criterion is not coordinate-free; it places extra emphasis 
upon reproducing the squared Euclidean distances of all N points from the origin. 
Our origin-centric perspective on affine reduction of dimensionality encourages inter¬ 
pretation of the recovered scatter using either polar or Cartesian coordinates. 

In traditional principal component and coordinate approaches to reduction of di¬ 
mensionality, the origin is conventionally placed at the geometric centroid of the 
overall scatter by subtracting the observed arithmetic mean value from each given co¬ 
ordinate. This convention can, optionally, be retained in the origin-centric approach 
discussed here. On the other hand, the impact of details such as how the given coordi¬ 
nates were originally scaled and intercorrelated are minimized in this new, alternative 
approach. This is accomplished by placing the given data scatter in its affine invari¬ 
ant canonical form, Obenchain(1971, 1972a). Interestingly, this substitution makes 
reduction of dimensionality using conventional principal components analysis impos¬ 
sible; all one-dimensional orthogonal projections of the revised data have the same 
variance, and any two mutually orthogonal projections are uncorrelated. In partic¬ 
ular, this means that the ultimate impact of researcher initiatives, Glaeser(2006), 
in choice of included variables and of their initial scaling upon potential recovered 
conhgurations of low dimensionality is thereby greatly reduced. 

The format of the paper is as follows. In the next three Sections, we introduce basic 
notation and dehne terminology related to Euclidean distance calculations, matrix 
norms in linear algebra, and origin-centric coordinate concepts. Sections and 
then display two relatively simple numerical examples. We end this overview paper 
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with some general discussion points in Section The hnal four Sections, to 11 
are somewhat more technical Appendices covering historical background information 
and some relatively advanced observations and conjectures. 


2 Origin-Centric Coordinates and Euclidean Inter¬ 
point Distances 


When a N X p matrix of hnite real numbers, X, is used as a Cartesian coordinate 
representation of N points in Euclidean space relative to p = 2 or more mutually 
orthogonal axes, the conventional view is that the location of the origin is not impor¬ 
tant. After all, the Pythagorean theorem dictates that the squared distance between 
the i-th and j-th points in the scatter is given by 




x')(xi 


= x'x, + x'x^ - 2x'x^, 


( 2 . 1 ) 


where x' denotes the i-th row of X. In the hrst line of the above expression, note that 
only differences in coordinates between points are involved. It follows that is, by 
dehnition, invariant under shifts in the location of the origin resulting from simple 
(additive) translations of coordinates. 


The second line of equation (2.1) shows an equivalent expression containing origin¬ 


centric terms. The hrst two terms are the squared Euclidean distances of the i-th and 
j-th points from their current origin, 0. 

Similarly, the full N x N matrix of inter-point squared distances can be written 
as 


= ((d^j)) ...for all i and j from 1 to N 
= (d^l' + l(dg)' - 2XX', 


( 2 . 2 ) 


where is the N x 1 column vector that consists of the ordered elements of 
Diag(XJi.'). In other words, the i-th element of dg is the x'xj term of equation 
(2.1). Thus dg is the vector of squared Euclidean distances of all N points from their 
current origin, 0. The matrix is symmetric, and its N diagonal elements are null. 

Suppose now that a given X matrix is to be optimally approximated by a N x q 
matrix of hnite real numbers, Z, where q < p but q is at least 1. Specihcally, suppose 
that the objective is to minimize the squared Frobenius norm of the diherence between 
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the matrices computed from the X and Z coordinates: 


N N 

Norm^ = " 4(Z))' 

1=1 j = l 

= ||pl' + lp'-2(XX'-ZZ')||^ 

= 4| |XX' - ZZ'I |2 + 2{p'lf + 2Np'p, (2.3) 


where p = do(^) ~ column vector of discrepancies in squared difference 

of points from the origin in X and Z coordinates. 

In the special case of primary interest here, the XX' outer-products matrix (iV x 
N) is what is know as an Association matrix, A, for “individuals” (points), Gower(1966), 
while the X'X inner-products matrix (p x p) contains the corresponding Adjusted 
Sums-of-Squares and Cross-Products for “variables.” The objective in Principal Com¬ 
ponent Analysis (PCA) is to minimize ||A — ZZ'|p where the X and Z matrices have 
each been “centered” as outlined below in Section This particular point is more 
fully discussed in an Appendix; see Section _ 

In any case, minimizing Norm^ of equation (2.3) is potentially quite different from 
ordinary PCA. Minimizing Norm^ places emphasis upon balancing two separate 
criteria: [ 1 ] reducing the size of the p vector, which contains the discrepancies in 
squared distance from the origin for all N points within the original X and derived 
Z configurations, and [2] reducing the PCA norm of the difference in outer-product 
matrices. Tension between t hese competing objectives make choice of Z so as to 
minimize Norm^ of equation (2.3) an origin-centric approach. 


3 Affine Reduction of Multivariate Dimensionality 


The Singular Value Decomposition of a N x p matrix consisting of the “centered,” 
real valued X coordinates of N points in a Euclidean space of dimension (linear rank) 
r can be written as: 


(I - l7')X = HA''“^G', 


(3.1) 


where the 1 x N row vector 7 ' is any generalized inverse of the N x 1 column vector 
of all ones, 1, the semi-orthogonal matrix H is the N x r matrix of standardized 
Principal Coordinates, A^/^ is a r x r diagonal matrix of strictly positive singular 
values, and G' is a r x p semi-orthogonal matrix of component Direction Cosines. 
When r = p, the G matrix becomes orthogonal (and invertible); GG' = I of order p. 

Every choice for 7 is such that 7 '! = 1; it follows from (3.1) that the semi- 
orthogonal matrix H is orthogonal to 7 . This 7 'H = 0 restriction assures that r 
< min{p,N — 1). More complete information about this critical choice of initial 
translation is given in the Appendix of Section choice of 7 determines the location 
of the origin, 0, of Cartesian coordinates. 
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Affine reductions in dimensionality are those in which the recovered Z configura¬ 
tion is of the restricted form: 

Z = HB, (3.2) 


where H is the N x r matrix of (3.1) and B is a r x q matrix of rank{Q) = q < r 
consisting of real scalars. In particular, any row of H that is null assures that the 
corresponding row of Z will also be null ...corresponding to a point coincident with 
the origin, 0. 

One of two forms of preliminary data “standardization” are typically applied prior 
to performing traditional PCA, but neither is nearly as “drastic” as replacing one’s 


initial X matrix by its Mahalanobis principal coordinates matrix, H, as in (3.2). 
Traditional “mean centering” uses the I"*" vector, with each of its N elements equal 
to 1/N, as choice of 7 ' in (3.1). Traditional standardization to “correlation form” 
performs an addition step in which each column of the “mean centered” X matrix is 
then divided by its sample standard deviation. This is essentially equivalent to the 
standardization in which the N elements in each column of X are made to not only 
sum to zero but also to have sum-of-squares set equal to 1 ; on the other hand, this 
standardization still allows the different columns of X to be (linearly) correlated or 
confounded. 


In sharp contrast, the semi-orthogonal H matrix of (3.1) is essentially r-dimensional 


from the perspective of traditional PCA. Each column of this H matrix is orthogonal 
to the current choice of 7 (defining a mean or multivariate median measure of loca¬ 
tion), has sum-of-squares set equal to 1 , and is exactly uncorrelated with the derived 
coordinates along any direction strictly orthogonal to it. 

For example, consider the 1-dimensional case, q = 1 in (3.2), where the Z and 
B matrices become single column vectors, z and b. It follows that 7 'z = 0 while 
the corresponding sum-of-squares is z'z = b'H'Hb = b'b, which is determined solely 
by the squared length of the b vector. Similarly, any two different choices for this 
b vector yield uncorrelated z representations if and only if these two choices are 
mutually orthogonal vectors, b'j^b 2 = 0 . 

In other words, since PCA cannot be used to reduce the dimensionality of H, 
one’s primary hope may well be to adopt an origin-centric perspective based upon 


minimizing the squared norm of equation (2.3). 


4 Adding a Point at the Origin 


If none of the rows of the H matrix of ( 3.1[ ) is null, then the given scatter contains no 
points coincident with the origin, as determined by the current choice of 7 vector. On 
the other hand, there will always be at least one point at the current origin whenever 
exactly one element of the current 7 vector is a 1 and all other elements are null. 


Any null row of the H matrix of (3.1) can be relocated to its first row simply by 


renumbering the N points in the initial scatter. The resulting d^X.) vector of (2.3) 
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that contains the sqnared Enclidean distances of all N point from their cnrrent origin 
will then natnrally have a zero in its first row. In fact, this dl(X.) vector will also 


then be the first colnmn of the matrix of ( 2 . 2 ), while its transpose will be the 
Erst row of this same matrix. 


When a H matrix from (3.1) contains no nnll row, a nnll row can be added as. 


say, row N+1. This augmented H matrix is still semi-orthogonal, and H'H is still the 
r X r identify matrix. The corresponding augmented 7 column vector {{N -|- 1) x 1) 
can be taken to have a zero as its hnal element, and the augmented matrix of 


( 2 . 2 ) will have the original vector as the above the diagonal elements in its last 

column ...and its transpose in the bottom row. 

In summary, the do(^) vector that contains the squared Euclidean distances of 
all N original points from their current origin is ei ther already contained within the 
original N x N squared distances matrix, of (2.2), or else it will appear within 


the corresponding (N-|-l) x (N-l-1) augmented version of this matrix when a new 
point coincident with the origin is included. 

In other words, the (ig(X) vector is eit her an explicit or implicit part of every N 
X N matrix of squared dista nces, of (2.2). In sharp contrast with PCA, efforts 
to minimize Norm^ of (2.3) place special emphasis upon accurate reproduction of 


dl(X.) values within the recovered Z-configuration. 


5 First Numerical Example: Six Points on a Plane 

Perhaps the most simple sort of “interesting” example for comparing PCA with our 
origin-centric approach is depicted as H coordinates (p=2 and N= 6 ) in Figure 1. 
When reducing this conhguration to a single dimension (q=l), a singular affine trans¬ 
formation can produce either “a pair of triplet points” or else a “triplet of pairs.” In 
fact, these particular conhgura tions happen to correspond to two of the three local 
minima of Norm^ of equation (2.3); see Figure 2. 


This example is also interesting in the sense that, while the initial H-configuration 
contains no point (s) at the origin, the optimal q=l dimensional recovered configura¬ 
tion [case (c) in Figure 2] contains 2 points coincident with the origin. 


6 Second Numerical Example: The Longley Data 
in q=2 Dimensions 


Here we use the infamous Longley(1972) dataset (p= 6 , N=16) to illustrate some 
key distinctions between the PCA and affine, origin-centric methods of (3.2) for re¬ 
duction of multivariate dimensionality. While Longley used seven variables in his 
ill-conditioned regression model, here we use YEAR = 1947 to 1962 only as labels 
for points, rather than as an exact “linear trend” variable. The six US economic 
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variables used here are Xi = GNP deflator, X 2 = GNP, X 3 = number Unemployed, 
X 4 = size of Armed Forces, X 5 = total Population and Xq = total Employed. 

This example is rather typical of cases I have explored, especially when N greatly 
exceeds p. Specifically, the optimal affine reduction in Mahalanobis dimensionality 
depicted in Figure 4 has a “Dyson swarm” interpretation, featuring a characteristic 
“hole” surrounding the explicit origin. 

7 Discussion 

In the terminology of Glaeser(2006), “Researcher Initiatives” can deliberately bias 
findings towards the perspective of one particular subset of stakeholders. For exam¬ 
ple, it is well know that including several different quantitative X-variables that are 
surrogate measures of a single construct in a PGA tends to orient the first principal 
axis to represent that construct. The corresponding phenomena in an origin-centric 
analysis would be to include several points that are actually coincident, thereby plac¬ 
ing greater emphasis upon recovering the distance of this composite from all other 
points within the scatter. 

Origin-centric methods allow the scatter to be visualized much like a “Dyson 
Swarm” for a finite universe of N or fewer points. For example, a q=3 dimensional 
recovered scatter might be primarily viewed as a 2 -dimensional small-scale conformal 
mapping where radial distance information is disregarded. Alternatively, the size 
of each of the N plotting symbols displayed on this sort of “sky view” map could 
represent some monotone function of its recovered radius. 

Nonnegative Matrix Factorization (NMF) techniques rely upon locating the origin 
so that all N initial points have p strictly nonnegative coordinates. In other words, 
all initial points must lie on or within the boundaries of the “first” Gartesian 2^- 
ant of p-dimensional Euclidean space. Similarly, the q dimensions to be recovered 
using NMF are to be represented as vectors emanating from the origin that have 
strictly nonnegative direction cosines because they point exclusively into this same 
first Gartesian 2^-ant. 


8 Appendix: Affine Invariance and Commutativ¬ 
ity 

Linear algebra is typically used to manipulate data structures, consisting of vectors 
and matrices of real numbers, in multivariate statistical analyses. Insights into the 
analytical geometry of such manipulations is provided by visualizing the data as 
consisting of a scatter of N points within a p-dimensional Euclidean space. 

A fundamental property of Euclidean space is that distance is computed from 
coordinates relative to orthogonal axes using the Pythagorean theorem. 
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A N X p matrix of real numbers, Y, is said to represent a nonsingular affine 
transformation of an initial N x p matrix of real numbers, X, if and only if there 
exists a 1 X p “translation” vector a' and a nonsingular p x p matrix B of real values 
such that 

Y = la' + XB. (8.1) 

In particular, if any row of X consists of all zeros, the corresponding row of Y will 
be a'. The notation of equation (8.1) differs slightly from that of Obenchain(1971) 


simply because the X and Y matrices have been transposed here. 

A statistic S{Y), consisting of a scalar, vector or matrix of real numbers, will 
be said to be affine invariant if and only if its numerical value when computed after 
any nonsingular affine transformation has been applied to a dataset is identical to its 
value before such transformation: 


S'(Y) = S'(X) ...for every translation a and nonsingular B 


( 8 . 2 ) 


The H matrix of (3.1) is not uniquely determined. For example, any of the r 


columns of H (each N x 1) can always be multiplied by —1 if the corresponding row 
of G' (1 X p) is also multiplied by —1. On the other hand, HH' is an idempotent 
and symmetric (N x N) matrix that is uniquely determined and corresponds to the 
orthogonal projection matrix (linear operator) that characterizes the column space 
of the (I — I 7 OX matrix; see Rao(1973), pages 46-48. In fact, Obenchain(1971) 
showed that HH' and are equivalent “maximal affine invariant” statistics which 
can be viewed as a standardized, canonical form for a given (N x p) data matrix of 
“centered” scalar values subject to nonsingular linear B transformations, (8.1). 


The HH' and matrices are unique but distinct, and each can be computed 
from the other. PGA focuses upon reproducing (I —l 7 ')XX'(I — 7 I') by decomposing 
it into ordered and additive components. PGA fails to reduce dimensionality when 
applied to HH' because all of its components are equally good or bad and, thus, 
cannot be uniquely ordered. 

By focusing upon reproducing instead of HH', origin-centric methods can 
reduce the dimensionality of conhgurations that PGA cannot. There appear to be 
at least two separate parts to the price-one-has-to-pay to achieve affine invariance 
properties that ignore vagaries caused by analyst choice of variable scalings and/or 
confounding between variables. First of all, recovered configurations will not be ad¬ 
ditive; the optimal solution of dimension (q-l-l) usually does not exactly “contain” 


the optimal solution of dimension q. Secondly, minimizing Norm^ of equation (2.3) 
is a problem of constrained optimization subject to multiple local minima. Fortu¬ 
nately, modern numerical search software, such as the “optimx” R-package of Nash 
and Varadhan(2011), is now available to researchers. 






9 Appendix: Detail on Data Centering and Affine 
Median Vectors 

A N X p matrix of real numbers, Y, will be said to be a “centered” version of a N x 
p matrix of real numbers, X, if and only if 

Y = (I-iy)X, (9.1) 


where the 1 x N row vector 7 ' is any generalized inverse of the N x 1 column vector 
of all ones, 1. In other words, any 1 x N row vector of real values that sum to 1, 
y'l = 1, is a valid choice for 7 ' in (9.1). 

The “centering” operation of equation (9.1) frequently yields Y coordinates that 
differ from those of the initial X matrix. In these cases, the location of the implicit 
origin, 0, has been shifted via a simple “translation” of Y coordinates. 

The unique Moore-Penrose inverse of 1, usually denoted by 1^, is the row vector 
with each of its N entries equal to 1/N. This choice for 7 ' corresponds to centering 
the X matrix at its overall mean vector, i.e. at its traditional centroid. 

When 7 i and 72 are two possibly different choices for the 7 vector of (9.1), the 
following equality is easily verihed by direct multiplication and algebraic simplihca- 
tion: 

(I- 17^(1-170 =1-172- (9.2) 

When 7 i = 72 , equation (9.2) shows that all valid choices for the 7 vector of ( 9.1[ ) yield 
a centering matrix, I — 17 ', that is idempotent and, thus, corresponds to a geometrical 
projection in N space that generally is oblique. In fact, the choice 7 j = 72 = I'*' is 
the only choice that makes the I — ll’*' matrix symmetric as well as idempotent, thus 
corresponding to a strictly orthogonal projection in N space. 

Perhaps, an even more interesting result follows from equation (9.2) when 71 and 
72 are distinct generalized inverses. In these cases, note that any valid, hnal choice 
of centering, as in equation (9.1), will simply “wipe-out” and “replace” any and all 
previously applied choices of centering of form (9.1). The corresponding implication 
for (3.1) is that changing one’s choice of 7 vector typically changes the H matrix so 
that it becomes orthogonal to the new 7 . 

To center an X matrix at the point corresponding to its i-th row, the i-th element 
of 7 would then be a 1 and all other elements would be null. 

A multivariate median vector can be dehned, as in Obenchain(1972a), using “con¬ 
vex hull” concepts to assure that the 7 vector is an invariant function under all strictly 
nonsingular affine transformations of a given X scatter, i.e. where no systematic re¬ 
ductions in dimensionality are being enforced. In direct analogy with the concept 
of a univariate median, successive “exterior” convex hulls (which are preserved un¬ 
der nonsingular affine transformations) of the scatter initially spanning p-dimensional 
Euclidean space would be determined and successively “peeled away.” Whenever two 
or more points on an exterior hull are coincident, only one such point is set aside 


9 













(assigned a 7 weight of zero) at each stage of such peeling. The hnal convex hull re¬ 
maining at the end of this sequence will contain neither any coincident points nor any 
strictly interior points. The affine invariant 7 vector, which dehnes the corresponding 
affine commutative median vector, 7 'X = —a' of equation (8.1), gives equal, positive 
weight to each of the unique points on the innermost hull and zero weight to all other 
points. 


10 Appendix: Restrictions to Distinct Points, pos¬ 
sibly Weighted. 

In this penultimate appendix, we comment on the role of mutually coincident points 
in determining the dimensionality of a given scatter. 

Since coincident points clearly cannot increase dimensionality and always remain 


coincident in the affine reduction formulation of (3.2), it makes good sense compu¬ 


tationally to limit attention to scatters of only distinct, non-coincident points. Since 
no reasonable metho d will fail to exactly reproduce the zero diag onal elements of the 
matrix of (2.2), minimizing the squared norm of equation (2.3) rightly focuses 


on minimizing the sum of the N{N — l)/2 terms with 1 < i < j < N, where N now 
denotes the total number of distinct points. On the other hand, individual terms in 
this summation could certainly be differently “weighted” to account for the presence 
or absence of additional points coincident with either the i-th or the j-th distinct 
point. 

Limiting attention to unique points and affine reduction methods also assures, 
when many variables are analyzed for a hxed number of points, that the maximum 


possible value of r will indeed equal {N — 1) < p in equation (3.1). For an arbitrary 
conhguration of N unique points, one then knows in advance that H'H will ultimately 
become the {N — 1) x {N — 1) identity matrix as the number of columns of X is 
arbitrarily increased. Furthermore, in this same limit, the corresponding HH' matrix 
will also approach the I — yy’*' matrix, which is the N x N orthogonal projection 
matrix for the vector space orthogonal to 7 . 


11 Appendix: Affine Invariance and the Restric¬ 
tion to p less than or equal to N — 1. 

When y' = 1+, the large-p asymptotic H-con£guration is that of N points equally 
spaced in V — 1 dimensions, which corresponds to a sparse, high-dimensional Dyson 
sphere with no point coincident with the origin. Specihcally, the squared distance 
between any 2 of these N points will be 2, and the squared distance from each to the 
origin will be = {N — 1)/N. 
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When all of the elements of 7 except one are zeros, the asymptotic conhguration 
will depict that single point as being coincident with the origin while the remaining 
N — 1 points are again equally spaced. While the squared distance between any two 
of these iV — 1 points will still be 2, the squared distance from each of them to the 
origin point will now be dg = 1. The first — 1 rows of the asymptotic H matrix 
may then be any (A^ — 1 ) x {N — 1 ) orthogonal matrix (such as some permutation of 
the rows and columns of the I matrix), while the final row is null. This configuration 
again spans A^ — 1 dimensions and depicts another sparse, high-dimensional Dyson 
sphere as well as its origin point. 
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Figure 1: Mahalanobis H Configuration for 6 Points in 2 Dimensions. Numerical 
searches for the best q=l representation can use, as their starting points, orthogonal 
projections of the 6 points onto the line through the origin making angle (j) with the 
horizontal axis. Initial values for the B matrix of (3.2) are then simply 2-element 
column vectors of the form (sm(0), cos(0))', and the objective of the search is to hnd 
the best length for B in each given direction. 
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Figure 2: Depiction of three Local Minima of Norm^ for Dimension q=l. While 
a pair of local minima occur in case (a) at 3.50 and in case (b) at 4.80, the global 
minimum of 3.11 corresponds to case (c). 


14 



Figure 3: The PCA solution of Dimension q=2 for the Longley data when standard¬ 
ized to “correlation” form. It is clear from the labels on the points as well as from 
the direction cosines for Axes 1, 2, 5 and 6 that this recovered conhguration is best 
interpreted as representing “linear growth” over time. 
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Figure 4: This is the optimal origin-centric representation in q=2 dimensions for the 
Longley data when standardized to Mahalanobis H form. The “year” labels on points 
suggest that time variation is being depicted here as clockwise or counter-clockwise 
rotations. In particular, the direction cosines for Axes 3 and 4 suggest that these 
two measures are the true, primary determinants of year-to-year variation in q=2 
dimensions. 
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